Apparatus for predicting metadata of medical image and method thereof

ABSTRACT

This disclosure relates to a computerized method to perform a machine learning on a relationship between medical images and metadata using a neural network and acquiring metadata by applying a machine learning model to medical images, and a method thereof. The apparatus and method may include training a prediction model for predicting metadata of medical images based on multiple medical images for learning and metadata matched with each of multiple medical images and predicting metadata of input medical image.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/708,205, filed Dec. 9, 2019, which claims priority to and the benefitof Korean Patent Application No. 10-2019-0059860, filed on May 22, 2019.The disclosure of each of these applications is incorporated herein byreference in its entirety.

BACKGROUND 1. Technical Field

This disclosure generally relates to an apparatus for performing machinelearning on relationship between a medical image and metadata using aneural network and acquiring metadata by applying a machine learningmodel to the medical image.

2. Discussion of Related Technology

According to digital imaging and communications and medicine (DICOM)which is a data standard for medical images, DICOM data largely containtwo types of information. One is an original medical image taken (rawpixel array), and the other is metadata recorded on a DICOM header.

During a medical image analysis, values recorded on the DICOM header areused first. For instance, medical workers determine whether a medicalimage corresponds to patient's body part by looking at“BodyUnitExamined” attribute of the DICOM header before interpreting themedical image. In addition, medical workers can normalize originalimages that come from diverse environments using “Window Center/Width”attribute of the DICOM header.

SUMMARY

Each hospital has a different protocol for metadata of such medicalimages saved on the DICOM header, and a different subjective value maybe entered by each radiologist. The DICOM header may not have values,have incorrect values, or have values saved according to differentcriteria. In this case, medical workers cannot interpret medical imagesor might interpret them incorrectly. In addition, normalized medicalimages are required to learn medical images by machine learning, butmachine learning cannot be performed properly on medical images if theirmetadata are saved according to different criteria.

A computerized method of analyzing medical images according to anembodiment of this disclosure comprises learning a prediction model forpredicting metadata of an input medical image based on multiple medicalimages (training medical images) and metadata matched with each of themultiple medical images and predicting metadata of the input medicalimage using the learned prediction model.

The metadata in the method of analyzing medical images according to anembodiment of this disclosure comprises at least one of informationrelated to objects included in the medical image, information (e.g., oneor more parameters) about shooting environment of the medical image, andinformation related to display method of the medical image.

In the method of analyzing medical images according to an embodiment ofthis disclosure, information related to objects included in the medicalimages comprises at least one of information about body parts includedin the medical images and information about patient, information aboutshooting environment of the medical images comprises at least one ofmodality information of medical images and information about shootingmethod of the medical images, and information related to display methodof the medical images comprises at least one of window centerinformation of the medical images, window width information, colorinversion information, image rotation information, and image flipinformation.

In the method of analyzing medical images according to an embodiment ofthis disclosure, learning a prediction model for the medical imageanalysis method comprises acquiring multiple metadata that match each ofmultiple medical images from standard data elements of DICOM header ofeach of multiple medical images and learning the prediction model usingmultiple medical images for learning and multiple metadata acquired.

In the method of analyzing medical images according to an embodiment ofthis disclosure, learning a prediction model additionally comprisesmatching and saving metadata predicted for input medical image with theinput medical image.

In the method of analyzing medical images according to an embodiment ofthis disclosure, saving the metadata comprises saving the predictedmetadata on DICOM header of the input medical image.

The method of analyzing medical images according to an embodiment ofthis disclosure additionally comprises adjusting the input medical imagebased on the predicted metadata to detect anomaly in the input medicalimage.

In the method of analyzing medical images according to an embodiment ofthis disclosure, adjusting the input medical image comprises adjustingat least one of window center of the input medical image, window width,color, and output direction.

In the method of analyzing medical images according to an embodiment ofthis disclosure, the multiple medical images for learning and the inputmedical image are images that correspond to DICOM standard.

An apparatus for analyzing medical images according to an embodiment ofthis disclosure comprises a processor and memory. The processor usesinstructions stored on the memory to execute training a prediction modelfor predicting metadata of the medical images based on multiple medicalimages and metadata matched with each of the multiple medical images andpredicting metadata of the input medical image using the trainedprediction model.

In the apparatus for analyzing medical image according to an embodimentof this disclosure, metadata includes at least one of informationrelated to objects included in the medical image, information aboutshooting environment of the medical image, and information related todisplay method of the medical image.

In the apparatus for analyzing medical image according to an embodimentof this disclosure, information related to objects included in themedical image comprises at least one of information body parts includedin the medical image and information about patient, information aboutshooting environment of medical image comprises at least one of modalityinformation of the medical image and information about shooting methodof the medical image, and information related to display method of themedical image comprises at least one of window center information,window width information, color inversion information, image rotationinformation, and image flip information of the medical image.

In the apparatus for analyzing medical image according to an embodimentof this disclosure, the processor uses instructions stored on the memoryto execute acquiring multiple metadata that match each of the multiplemedical images from standard data elements of DICOM header of each ofthe multiple medical images and training the prediction model using themultiple medical images for training and multiple metadata acquired.

In the apparatus for analyzing medical image according to an embodimentof this disclosure, the processor uses instructions stored on the memoryto further execute matching and saving the metadata predicted for theinput medical image with the input medical image.

In the apparatus for analyzing medical image according to an embodimentof this disclosure, the processor uses instructions stored on the memoryto further execute saving the predicted metadata on DICOM header of theinput medical image.

In the apparatus for analyzing medical image according to an embodimentof this disclosure, the processor uses instructions stored on the memoryto further execute adjusting the input medical image based on thepredicted metadata to detect anomaly in the input medical image.

In the apparatus for analyzing medical images according to an embodimentof this disclosure, the processor uses instructions stored on the memoryto further execute adjusting at least one of window center, windowwidth, color, and output direction of the input medical image based onthe predicted metadata.

in the apparatus for analyzing medical image according to an embodimentof this disclosure, the multiple medical images for training and inputmedical image are images that correspond to the DICOM standard.

In addition, a program to embody the method of analyzing medical imagecan be recorded on a computer readable recording medium.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects will become apparent and more readilyappreciated from the following description of embodiments, taken inconjunction with the accompanying drawings.

FIG. 1 is a block diagram of a medical image analysis apparatusaccording to an embodiment of this disclosure.

FIG. 2 is a figure that represents the medical image analysis apparatusaccording to an embodiment of this disclosure.

FIG. 3 is a flow diagram that illustrates operation of the medical imageanalysis apparatus according to an embodiment of this disclosure.

FIG. 4 is a figure that illustrates structure of the DICOM fileaccording to an embodiment of this disclosure.

FIG. 5 shows CT information based on window center information andwindow width information according to an embodiment of this disclosure.

FIG. 6 is a flow diagram that illustrates operation of the medical imageanalysis apparatus according to an embodiment of this disclosure.

FIG. 7 is a figure that illustrates a prediction model learning processaccording to an embodiment of this disclosure.

FIG. 8 is a flow diagram that illustrates operation of the medical imageanalysis apparatus according to an embodiment of this disclosure.

FIG. 9 is a figure that illustrates a process of using the predictionmodel according to an embodiment of this disclosure.

FIG. 10 is a flow diagram that illustrates a process of detecting lesionaccording to an embodiment of this disclosure.

DETAILED DESCRIPTION

Merits and characteristics of embodiments disclosed and methods ofachieving them would be clarified by referring to the embodimentsdescribed below with attached drawings. However, this disclosure is notlimited to the embodiments disclosed below and can be embodied intodiverse forms. These embodiments are simply provided to make thisdisclosure complete and to completely inform the scope of this inventionto persons with common knowledge in the technical field of thisdisclosure.

Terms used in this specification will be explained briefly. The termsused in this specification are ordinary terms that are used widely,selected in consideration of functions of this disclosure. These termscan change according to intention of engineers who work in relatedfields, precedents, appearance of new technologies, etc. In addition,certain terms were selected arbitrarily by the applicant, for which casemeanings of such terms will be explained in detail. Therefore, the termsused in this disclosure must be defined based on their definitions andoverall application in this disclosure instead of their names.

Unless clearly specified to be singular, singular expressions used inthis specification shall also include plurality. In addition, unlessclearly specified to be plural, plural expressions shall includesingularity.

When a part of this specification is said to “comprise” a component,this does not exclude other components and means that other componentscan also be included unless specifically described otherwise.

In addition, term “unit” used in this specification refers to a softwareor hardware component. A “unit” plays certain roles, but it is notlimited to software or hardware. A “unit” can exist in an addressablestorage medium or play one or more processors. Therefore, for instance,“units” include components such as software components, object-orientedsoftware components, class components and task components, processes,functions, attributes, procedures, subroutines, program code segments,drivers, firmware, microcode, circuit, data, database, data structures,tables, arrays, and variables. Functions provided within components and“units” can be combined into smaller number of components and “units” orsubdivided into additional components and “units.”

According to an embodiment of this disclosure, “units” can be embodiedusing a processor and a memory. The term “processor” is interpretedbroadly to include a general-purpose processor, a central processingunit (CPU), a microprocessor, a digital signal processor (DSP), acontroller, a microcontroller, a state machine, etc. in someenvironments, “processor” may refer to application specific integratedcircuit (ASIC), programmable logic device (PLD), field programmable gatearray (FPGA), etc. The term “processor” may, for instance, also refer tocombination of a DSP and a microprocessor, a combination of multiplemicroprocessors, a combination of one or more microprocessors combinedwith a DSP core, or combination of processing devices that is same asother combinations of such configuration.

The term “memory” is interpreted broadly to include a random electroniccomponent that can save electronic information. The term memory may alsorefer to various types of processor-readable medium such asrandom-access memory (RAM), read-only memory (ROM), non-volatilerandom-access memory (NVRAM), programmable read-only memory (PROM),erasable programmable read-only memory (EPROM), electrically erasablepROM (EEPROM), flash memory, magnetic or optical data storage device,registers, etc. Whereas a processor can read information from a memoryand record information on a memory, a memory is called to be inelectronic communication with a processor. A memory integrated with aprocessor is in electronic communication with the processor.

Certain embodiments of the present disclosure are explained in detailbelow by referring to attached figures so that this disclosure can beeasily implemented by persons with common knowledge in the technicalfield of this disclosure. To clarify explanation of this disclosure inthe figures, parts irrelevant to the explanation are omitted.

FIG. 1 is a block diagram of a medical image analysis apparatus (100)according to an embodiment of this disclosure.

Referring to FIG. 1, the medical image analysis device (100) accordingto an embodiment can comprise a data learning unit (110) and a datarecognition unit (120). The medical image analysis device (100) cancomprise a processor and memory.

The data learning unit (110) can learn a machine learning model using adata set to perform a target task. The data learning unit (110) canreceive label information related to the data set and target task. Thedata learning unit (110) can acquire the machine learning model byperforming machine learning on a relationship between the data set andlabel information. The machine learning model acquired by the datalearning unit (110) can be a model to generate the label informationusing the data set.

The data recognition unit (120) can receive and save the machinelearning model of the data learning unit (110). The data recognitionunit (120) can output the label information by applying input data tothe machine learning model. In addition, the data recognition unit (120)can be used to renew the machine learning model using the input data,label information, and output from the machine learning model.

At least one of the data learning unit (110) and data recognition unit(120) can be made into at least one hardware chip and mounted onelectronic apparatus. For instance, at least one of the data learningunit (110) and data recognition unit (120) can be made into an exclusivehardware chip for artificial intelligence (AI) or as a unit of anexisting general-purpose processor (e.g. CPU or application processor)or graphic-only processor (e.g. GPU) to be mounted on various electronicapparatus explained earlier.

In addition, the data learning unit (110) and data recognition unit(120) can be mounted separately on different electronic apparatuses. Forinstance, one of the data learning unit (110) and the data recognitionunit (120) can be included in an electronic apparatus with the other oneincluded in a server. In addition, the data learning unit (110) and datarecognition unit (120) can be connected with or without wire to providethe machine learning model constructed by the data learning unit (110)to the data recognition unit (120) or provide the input data of the datarecognition unit (120) to the data learning unit (110) as additionallearning data.

On the one hand, at least one of the data learning unit (110) and datarecognition unit (120) can be embodied into a software module. If atleast one of the data learning unit (110) and data recognition unit(120) is embodied into a software module (or a program module thatincludes instructions), the software module can be saved on the memory,or non-transitory computer readable media. In addition, in this case, atleast one software module can be provided by an operating system (OS) orby a prescribed application. Otherwise, at least one software module canhave a portion provided by an OS and the other portion provided by aprescribed application.

The data learning unit (110) according to an embodiment of thisdisclosure can include a data acquisition unit (111), a preprocessingunit (112), a learning data selection unit (113), a model learning unit(114), and a model evaluation unit (115).

The data acquisition unit (111) can acquire data necessary for machinelearning. Since a large volume of data is needed for learning, the dataacquisition unit (111) can receive data sets that include multiple data.

The label information can be assigned to each of multiple data. Thelabel information may be information that explains each of multipledata. The label information may be information to be derived by thetarget task. The label information can be acquired from user input,memory, or result of the machine learning model. For instance, if thetarget task is to determine existence of a certain object in an image,multiple data would be multiple image data and the label informationwould be whether the certain object exists in each of multiple images.

The preprocessing unit (112) can preprocess acquired data so that datareceived can be used for machine learning. The preprocessing unit (112)can process the acquired data sets into a preset format to be used bythe model learning unit (114) to be described later.

The learning data selection unit (113) can select data necessary forlearning among preprocessed data. Selected data can be provided to themodel learning unit (114). The learning data selection unit (113) canselect data necessary for learning among preprocessed data according tothe preset standards. In addition, the learning data selection unit(113) can also select data according to the preset standards throughlearning of the model learning unit (114) to be described later.

The model learning unit (114) can learn standards for label informationoutput based on the data set. In addition, the model learning unit (114)can perform machine learning by using the data set and label informationof the data set as learning data. In addition, the model learning unit(114) can perform machine learning by additionally using the acquiredmachine learning model. In this case, the acquired machine learningmodel can be a model constructed in advance. For instance, the machinelearning model can be a model constructed in advance by receivingdefault learning data.

The machine learning model can be constructed by considering applicationfield of the learning model, purpose of learning, computer performanceof the apparatus, etc. The machine learning model, for instance, can bea model based on a neural network. For example, models like deep neuralnetwork (DNN), recurrent neural network (RNN), long short-term memorymodels (LSTM), bidirectional recurrent deep neural network (BRDNN), andconvolutional neural networks (CNN) can be used as machine learningmodels, but the machine learning model is not limited to them.

According to various embodiments, if there are multiple machine learningmodels constructed in advance, the model learning unit (114) can decidea machine learning model that is highly associated with input learningdata and default learning data as the machine learning model to belearned. In this case, default learning data can be already classifiedinto data types, and the machine learning model can be constructed inadvance for each data type. For instance, default learning data can beclassified in advance according to various criteria including placewhere learning data are generated, time at which learning data aregenerated, size of learning data, a learning data generator, an objecttype of learning data, etc.

In addition, the model learning unit (114), for instance, can learn themachine learning model using a learning algorithm that includes errorback-propagation or gradient descent.

In addition, the model learning unit (114), for instance, can learn themachine learning model through supervised learning that uses learningdata as input values. In addition, the model learning unit (114), forinstance, can acquire the machine learning model through unsupervisedlearning that finds criteria for the target task by learning a data typeneeded for the target task on its own without supervision. In addition,the model learning unit (114), for instance, can learn the machinelearning model through reinforcement learning that uses feedback oncorrectness of the result of the target task according to learning.

In addition, once the machine learning model is learned, the modellearning unit (114) can save the learned machine learning model. In thiscase, the model learning unit (114) can save the learned machinelearning model on the memory of electronic apparatus that includes thedata recognition unit (120). Otherwise, the model learning unit (114)can also save the learned machine learning model on the memory of theserver connected to the electronic apparatus connected via wired orwireless network.

The memory that saves the learned machine learning model, for instance,can also save commands or data related to at least one other componentof electronic apparatus. In addition, the memory can save softwareand/or programs. Programs, for instance, may comprise kernel,middleware, application programming interface (API) and/or applicationprogram (or “application”), etc.

The model evaluation unit (115) can enter evaluation data into themachine learning model and make the model learning unit (114) repeatlearning if the output results from evaluation data fail to satisfyprescribed criteria. In this case, evaluation data may be preset data toevaluate the machine learning model.

For instance, in the results of the machine learning model learned forevaluation data, the model evaluation unit (115) can be evaluated as tonot satisfy the prescribed criteria if the number or ratio of evaluationdata with inaccurate recognition result exceeds the preset thresholdvalue. For example, if the prescribed criteria are defined as ratio of2% and the learned machine learning model outputs incorrect recognitionresult for 20 evaluation data out of 1,000 evaluation data, the modelevaluation unit (115) can evaluate that the learned machine learningmodel is inappropriate.

On the one hand, if there are multiple learned machine learning models,the model evaluation unit (115) can evaluate whether each of the imagelearning model satisfies the prescribed criteria and decide the modelthat satisfies the prescribed criteria as the final machine learningmodel. In this case, if multiple models satisfy the prescribed criteria,the model evaluation unit (115) can decide one or prescribed number ofmodels preset according to evaluation score as the final machinelearning model.

On the one hand, at least one of the data acquisition unit (111), thepreprocessing unit (112), the learning data selection unit (113), themodel learning unit (114), and the model evaluation unit (115) in thedata learning unit (110) can be made into at least one hardware chip andmounted on the electronic apparatus. For instance, at least one of theunits 111-115 can be made into an exclusive hardware chip for artificialintelligence (AI) or be made into a unit of an existing general-purposeprocessor (e.g. CPU or application processor) or graphic-only processor(e.g. GPU) and mounted on various electronic apparatus describedearlier.

In addition, the data acquisition unit (111), the preprocessing unit(112), the learning data selection unit (113), the model learning unit(114), and the model evaluation unit (115) may be mounted on anelectronic apparatus or separately on different electronic apparatuses.For instance, the units 111-115 may have some of them included in theelectronic apparatus and others in the server.

In addition, at least one of the units 111-115 can be embodied into asoftware module. if at least one of the data acquisition unit (111), thepreprocessing unit (112), the learning data selection unit (113), themodel learning unit (114), and the model evaluation unit (115) isembodied into a software module (or a program module that includesinstructions), the software module can he saved on non-transitorycomputer readable media. In addition, in this case, at least onesoftware module can be provided by an OS or by a prescribed application.Otherwise, at least one software module can have a portion provided byan OS and the other portion provided by a prescribed application.

The data recognition unit (120) according to an embodiment of thisdisclosure may include a data acquisition unit (121), a preprocessingunit (122), a recognition data selection unit (123), a recognitionresult provision unit (124), and a model renewal unit (125).

The data acquisition unit (121) can receive input data. Thepreprocessing unit (122) can preprocess input data acquired so thatinput data acquired is used by the recognition data selection unit (123)or recognition result provision unit (124).

The recognition data selection unit (123) can select necessary dataamong preprocessed data. Selected data can be provided to therecognition result provision unit (124). The recognition data selectionunit (123) can select a unit or all of preprocessed data according topreset criteria. In addition, the recognition data selection unit (123)can also select data according to the preset criteria through learningby the model learning unit (114).

The recognition result provision unit (124) can acquire result data byapplying selected data to the machine learning model. The machinelearning model can be a machine learning model generated by the modelearning unit (114). The recognition result provision unit (124) canoutput result data.

The model renewal unit (175) can renew the machine learning model basedon evaluation of a recognition result provided by the recognition resultprovision unit (124). For instance, the model renewal unit (125) canmake the model learning unit (114) renew the machine learning model byproviding a recognition result provided by the recognition resultprovision unit (124) to the model learning unit (114).

On the one hand, at least one of the data acquisition unit (121), thepreprocessing unit (122), the recognition data selection unit (123), therecognition result provision unit (124), and the model renewal unit(125) in the data recognition unit (120) can be made into at least onehardware chip and mounted on the electronic apparatus. For instance, atleast one of the units 121-125 can be made into an exclusive hardwarechip for artificial intelligence (AI) or made into a unit of an existinggeneral-purpose processor (e.g. CPU or application processor) orgraphic-only processor (e.g. GPU) and mounted on various electronicapparatus described earlier.

In addition, the units 121-125 can be mounted on one electronicapparatus or separately on different electronic apparatuses. Forinstance, the data acquisition unit (121), the preprocessing unit (122),the recognition data selection unit (123), the recognition resultprovision unit (124), and the model renewal unit (125) can have some ofthem included in the electronic apparatus and others in the server.

In addition, at least one of the units 121-125 can be embodied into asoftware module. If at least one of these units 121-125 is embodied intoa software module (or a program module that includes instructions), thesoftware module can be saved on a non-transitory computer readablemedia. In addition, in this case, at least one software module can beprovided by an OS or by a prescribed application. Otherwise, at leastone software module can have a portion provided by an OS and the otherportion provided by a prescribed application.

Method of sequential machine learning of data sets by the data learningunit (110) and apparatus thereof are explained in greater detail below.

FIG. 2 is a figure that represents the medical image analysis apparatusaccording to an embodiment of this disclosure.

The medical image analysis apparatus (200) can include a processor (210)and a memory (220). The processor (210) can execute instructions storedon the memory (220).

As described above, the medical image analysis apparatus (200) cancomprise at least one of a data learning unit (110) and a datarecognition unit (120). At least one of the data learning unit (110) anddata recognition unit (120) can be embodied by the processor (210) andmemory (220).

An operation of the medical image analysis device (200) is explained ingreater detail below.

FIG. 3 is a flow diagram that illustrates an operation of the medicalimage analysis apparatus according to an embodiment of this disclosure.

The medical image analysis apparatus (200) can execute a step (310) inwhich metadata of medical image is predicted using multiple medicalimages for learning and metadata matched with each of multiple medicalimages. A prediction model can be acquired by performing machinelearning on a relationship between a medical image and metadata based onthe data learning unit (110) of the medical image analysis apparatus(200). The prediction model can correspond to the machine learning modelin FIG. 1. The medical image analysis apparatus (200) can save theacquired prediction model on the memory or send it to another medicalimage analysis apparatus (200) via wired or wireless communication.

In addition, the medical image analysis apparatus (200) can execute astep (320) in which metadata of an input medical image is predictedusing the learned prediction model. The data recognition unit (120) ofthe medical image analysis apparatus (200) can predict metadata byapplying the prediction model to the input medical image. The predictionmodel can be acquired from the memory of the medical image analysisapparatus (200) or received from another medical image analysisapparatus (200).

Multiple medical images for learning and the input medical image can beimages of various formats.

For instance, multiple medical images for learning and the input medicalimage can be images that correspond to the DICOM standard. According tothe DICOM standard, the medical image analysis apparatus (200) can saveinformation related to medical images on the DICOM header.

The DICOM header can include standard data elements. Standard dataelements refer to elements related to medical images defined by theDICOM standard. The medical image analysis apparatus (200) can acquiremetadata from standard data elements. The DICOM header can includenon-standard data elements. Non-standard data elements are not definedby the DICOM standards, but they refer to elements related to medicalimages generated by a medical image apparatus manufacturer or medicalinstitution as needed. The medical image analysis apparatus (200) canacquire metadata from non-standard data elements.

Information related to medical images can be saved in storage spaceother than the DICOM header. The medical image analysis apparatus (200)can save diverse information related to medical images, along withmatching relationship of medical images. In addition, the medical imageanalysis apparatus (200) can acquire metadata based on diverseinformation related to medical images.

A process of acquiring metadata form the DICOM header is explained ingreater detail with FIG. 4 below.

FIG. 4 is a figure that illustrates a structure of the DICOM fileaccording to an embodiment of this disclosure.

A DICOM file (410) can comprise a DICOM header (411) and a medical image(412). The medical image (412) can include various medical images, forinstance at least one of CT, X-RAY, mammography or MRI image. The DICOMheader (411) can include diverse information related to medical images.The medical image analysis apparatus (200) can acquire metadata (420)based on diverse information related to the medical image (412) includedin the DICOM header (411).

The DICOM header (411) can comprise standard data elements ornon-standard data elements. The medical image analysis apparatus (200)can acquire metadata based on standard data elements or non-standarddata elements. Metadata (420) can comprise at least one of informationrelated to objects included in a medical image, information about ashooting environment of a medical image, and information related todisplay method of medical image.

More specifically, information related to objects included in a medicalimage can include at least one of information about body parts includedin the medical image and information about patient. Information aboutbody parts included in the medical image can be expressed as an indexthat corresponds to body parts. For instance, information about bodyparts can include at least one of indices indicating the lungs, abdomen,arms or legs.

Information about patients can comprise sex or age information ofpatients. Age information of patients can be a value that indicates ageof the patient as a number. In addition, metadata can comprise thebirthday of the patient, and the medical image analysis apparatus (200)can calculate age information of the patient using the birthday of thepatient. In addition, age information of the patient can be informationthat represents an age range, for instance an age group. As anembodiment, age information of the patient can be expressed as an indexthat indicates child, youth, middle age, old age, or age group.

Information about a shooting environment of medical images can comprisediverse information related to shooting of medical images. The shootingenvironment information can include at least one of modality informationof medical images and information about a shooting method of medicalimages.

Modality information of medical images can show type of imagingequipment used to shoot medical images. For instance, modalityinformation of medical images can be an index indicating that themedical image (412) is a CT, MRI, X-RAY, mammography, or ultrasonicimage. However, modality information of medical images is not limited tothese and can show diverse medical images taken on patients.

In addition, information about a shooting environment of medical imagescan comprise information about a shooting method of medical images. Theshooting environment information can correspond to a predefined indexthat is indicated as a number or text. The shooting environmentinformation can comprise information about whether an X-RAY wasirradiated from anterior to posterior of the patient or from posteriorto anterior or of the patient. In general, an X-RAY is irradiated fromposterior to anterior of the patient when the patient is standing up. Ifthe patient has difficulty standing up, an X-RAY is irradiated fromanterior to posterior.

Information related to a display method of medical images can compriseat least one of window center information of medical images, windowwidth information, color inversion information, image rotationinformation, and image flip information.

Window center information and window width information are explainedwith FIG. 5.

FIG. 5 shows CT information based on window center information andwindow width information according to an embodiment of this disclosure.

Window center information and window width information can beinformation to adjust brightness and contract of medical images.

A window graph can be drawn based on window center information (531) andwindow width information (532). Horizontal axis of window graph canrepresent input pixel values. An input pixel refers to a pixel of aninput medical image. Input pixel values can have a minimum value andmaximum value. Minimum and maximum values can be determined by at leastone of an image shooting device, an image display device, and imageencoding and decoding standards. If an input pixel value that has themaximum value can indicate brightest pixel, and an input pixel valuethat has the minimum value can indicate darkest pixel. However, pixelvalues are not limited to these indications.

Vertical axis of the window graph can represent output pixel values. Themedical image analysis apparatus (200) can determine output pixel valuesby processing input pixel values. The medical image analysis apparatus(200) can show medical image on display based on output pixel values.

For instance, the window graph (521) can be created if window centerinformation (531) is a and window width information (532) is b. Themedical image analysis apparatus (200) can generate a CT image (511)based on window center information (531) and window width information(532). The medical image analysis apparatus (200) can generate the CTimage (511) based on window center information (531) or window widthinformation (532) by indicating the input pixel value less than firstthreshold value as the minimum pixel value and indicating the inputpixel value greater than second threshold value as the maximum pixelvalue. In other words, the medical image analysis apparatus (200) canseparately indicate input pixel values that are greater than or equal tothe first threshold value and less than or equal to the second thresholdvalue.

The input pixel value less than the first threshold value or the inputpixel value greater than the second threshold value can be a cluttersignal unimportant for medical image analysis. The medical imageanalysis apparatus (200) can adjust the first threshold value and secondthreshold value based on window center information (531) and windowwidth information (532) and only indicate pixels that are important formedical image analysis.

In addition, for instance, the window graph (522) can appear as in FIG.5 if window center information (531) is a and window width information(532) is c. The medical image analysis apparatus (200) can generate a CTimage (512) based on window center information (531) and window widthinformation (532). The medical image analysis apparatus (200) cangenerate the CT image (511) based on window center information (531) orwindow width information (532) by separately indicating all input pixelvalues.

The medical image analysis apparatus (200) can indicate bright part ofinput pixels to be brighter or darker based on slope of the window graph(522). The medical image analysis apparatus (200) can adjust brightnessof the medical image based on window center information (531) or windowwidth information (532). For instance, comparing cases in which windowwidth information (532) is c and window width information (532) is b,the CT image (512) is darker than the CT image (511).

In comparison to the CT image (511), the CT image (512) includes allpixel values and does not lose information. However, since this imageexpresses all clutter signals that are unimportant for medical imageanalysis, it may not be optimized for image analysis. The medical imageanalysis apparatus (200) can optimize the medical image for imageanalysis by adjusting window center information (531) or window widthinformation (532).

In addition, for instance, the window graph (523) can appear as in FIG.5 if window center information (531) is d and window width information(532) is c. The medical image analysis apparatus (200) can generate a CTimage (513) based on window center information (531) and window widthinformation (532). The medical image analysis apparatus (200) cangenerate the CT image (513) based on window center information (531) orwindow width information (532) by processing all input pixel valuesgreater than third threshold value to be bright.

The medical image analysis apparatus (200) can indicate a bright part ofinput pixels to be brighter or darker using slope of the window graph(522). The medical image analysis apparatus (200) can adjust abrightness of the medical image based on window center information (531)or window width information (532). For instance, comparing cases inwhich window center information (531) is a and window center information(531) is d, the CT image (512) is darker than the CT image (513).

Input pixel values greater than the third threshold value can be cluttersignals that are unimportant for medical image analysis. The medicalimage analysis apparatus (200) can adjust the third threshold valuebased on window center information (531) and window width information(532) and only indicate pixels that are important for medical imageanalysis.

The medical image analysis apparatus (200) can normalize original imagesthat come from diverse environments based on window center information(531) or window width information (532). The preprocessing unit (112)and preprocessing unit (122) of the medical image analysis apparatus(200) can generate a normalized medical image from the original medicalimage. In addition, the medical image analysis apparatus (200) canprovide prediction model to another medical image analysis apparatus.Another medical image analysis apparatus can adjust the medical imagebased on the prediction model of this disclosure before executing othermachine learning cases.

Referring to FIG. 4 again, information related to a display method ofmedical images can comprise color inversion information. The medicalimage analysis apparatus (200) can invert color of the medical imagebased on color inversion information. If color inversion informationindicates inversion of color, the medical image analysis apparatus (200)can display the medical image by defining pixel value as value thatsubtracted the pixel value of the medical image from the maximum pixelvalue.

The display method information can include image rotation information.Image rotation information can show a size of clockwise orcounterclockwise rotation of the medical image taken. Image rotationinformation can be expressed as an index that corresponds to rotationsize or as a number in radian or degree. The medical image analysisapparatus (200) can rotate the medical image based on rotationinformation of the image.

The display method information can comprise image flip information.Image flip information can represent flipping of the medical image toleft and right about vertical axis. However, it is not limited to leftand right flip. Image flip information can represent flipping of themedical image up and down about horizontal axis.

Explanation so far was that metadata (420) includes at least one ofinformation related to objects included in medical images, informationabout shooting information of medical images, and information related todisplay method of medical images.

As described above, the medical image analysis apparatus (200) canacquire metadata based on information saved on the DICOM header in astandard format. In addition, the medical image analysis apparatus (200)can acquire metadata based on information saved on the DICOM header in anon-standard format. In addition, the medical image analysis apparatus(200) can acquire metadata based on information saved on storage spaceother than the DICOM header in a non-standard format.

Non-standard formats can differ among medical imaging devicemanufacturers or hospitals. If metadata is acquired from informationsaved in a non-standard format, the medical image analysis apparatus(200) can have an inconvenience of having to acquire metadata usingdifferent methods for different manufacturers or hospitals providing themedical image.

The medical image analysis apparatus (200) according to this disclosurecan generate metadata based on a medical image (412) even if metadata isacquired based on information saved in a non-standard format or there isno information related to the medical image. A step (310) of learningprediction model in FIG. 3 is explained in detail using FIG. 6 and FIG.7.

FIG. 6 is a flow diagram that illustrates an operation of the medicalimage analysis apparatus according to an embodiment of this disclosure.In addition, FIG. 7 is a figure that illustrates a prediction modellearning process according to an embodiment of this disclosure.

The medical image analysis apparatus (200) can receive an input data set(710) to learn the prediction model. The input data set (710) caninclude multiple medical images (711) and metadata (712).

The medical image analysis apparatus (200) can execute a step (610) inwhich multiple medical images (711) are acquired. For instance, themedical image analysis apparatus (200) can acquire multiple medicalimages from the memory (220). In addition, the medical image analysisapparatus (200) can acquire multiple medical images based on wired orwireless communication.

The medical image analysis apparatus (200) can execute a step (20) inwhich metadata (712) matched with each of multiple medical images isacquired. The medical image analysis apparatus (200) can execute a stepin which multiple metadata matched with each of multiple medical imagesare acquired from standard data elements of the DICOM header of each ofmultiple medical images for learning. However, acquisition is notlimited to this. The medical image analysis apparatus (200) can acquiremetadata from non-standard data elements of the DICOM header orinformation of a non-standard format saved on storage space other thanthe DICOM header. Redundant explanation about this is omitted because itis identical to explanation on FIG. 3 and FIG. 4.

The medical image analysis apparatus (200) can execute a step (630) inwhich the prediction model is learned using multiple medical images forlearning and multiple metadata acquired. The medical image analysisapparatus (200) can perform supervised learning using the originalmedical image and label data. Label data can be metadata. Label data canbe information on the DICOM header, information saved in an area otherthan the DICOM header, information entered by user, or information aboutoriginal medical image entered by a medical professional. The medicalimage analysis apparatus (200) can perform machine learning based onregression or classification according to characteristics of label data.

Machine learning can be used to learn the prediction model of themedical image analysis apparatus (200). Machine learning can be based onneural network. For instance, algorithms such as DNN, RNN, LSTM, BRDNN,and CNN can be used for machine learning, but machine learning is notlimited to them.

The medical image analysis apparatus (200) can output learning result asthe prediction model (730). The medical image analysis apparatus (200)can save the prediction model (730) on the memory, The medical imageanalysis apparatus (200) can send the prediction model (730) to anothermedical image analysis apparatus (200).

The step (310) of learning the prediction model was explained so far. Astep (320) in which metadata is predicted using the prediction model isexplained with FIG. 8 and FIG. 9 below.

FIG. 8 is a flow diagram that illustrates an operation of the medicalimage analysis apparatus according to an embodiment of this disclosure.In addition, FIG. 9 is a figure that illustrates a process of using theprediction model according to an embodiment of this disclosure.

The medical image analysis apparatus (200) can include a predictionmodel. The medical image analysis apparatus (200) can receive theprediction model from another medical image analysis apparatus (200). Inaddition, the medical image analysis apparatus (200) can acquire theprediction model by performing machine learning based on multiplemedical images and metadata.

The medical image analysis apparatus (200) can execute a step (810) inwhich a medical image (910) is received. For instance, the medical imageanalysis apparatus (200) can receive the medical image (910) as userinput through an input device. For another instance, the medical imageanalysis apparatus (200) can receive the medical image (910) fromanother apparatus via wired or wireless communication. The medical image(910) can be independent from multiple medical images (711). The medicalimage (910) can be different from or same as multiple medical images(711).

The medical image analysis apparatus (200) can execute a step (820) inwhich the prediction model is used to predict metadata (930) thatcorresponds to the input medical image (910). Predicted metadata (930)can comprise at least one of information related to objects included inthe medical image (910), information about a shooting environment of themedical image above, and information related to display method of themedical image (910).

As explained earlier, information related to objects included in medicalimages can include at least one of information about body parts includedin medical images and information about the patient. In addition, theshooting environment information can include at least one of modalityinformation of a medical images and information about a shooting methodof the medical image. In addition, information related to a displaymethod of medical images can include at least one of window centerinformation of the medical image, window width information, colorinversion information, image rotation information, and image flipinformation.

In addition, the medical image analysis apparatus (200) can execute astep (830) in which metadata (930) predicted for the input medical image(910) is matched with the input medical image (910) and saved. Themedical image analysis apparatus (200) can save predicted metadata (930)on the DICOM in a standard format, but it is not limited to the standardformat. The medical image analysis apparatus (200) can save predictedmetadata (930) on the DICOM header in a non-standard format or save iton storage space other than the DICOM header.

The medical image analysis apparatus (200) can adjust the input medicalimage to optimal condition or optimal state for target task. Forinstance, the medical image analysis apparatus (200) can execute anadditional step in which the input medical image is adjusted usingpredicted metadata (930) to detect anomaly in the input medical image.In addition, the medical image analysis apparatus (200) executes a stepin which at least one of window center of the input medical image,window width, color, and output direction is adjusted based on predictedmetadata (930).

For instance, predicted metadata (930) can comprise at least one ofpredicted window center information, predicted window width information,predicted color inversion information, predicted image rotationinformation, and predicted image flip information. The medical imageanalysis apparatus (200) can adjust window center or window width of themedical image (910) based on predicted window center information orpredicted window width information. In addition, the medical imageanalysis apparatus (200) can adjust color of the medical image (910)based on predicted color inversion information. In addition, the medicalimage analysis apparatus (200) can decide output direction of themedical image (910) based on predicted image rotation information andpredicted image flip information.

The medical image analysis apparatus (200) can predict metadata neededfor the original medical image before interpreting lesion from themedical image. In addition, the medical image analysis apparatus (200)can adjust the medical image to be interpretable based on the predictedvalue. In addition, interpretability of the medical image can bedetermined based on the predicted value. Therefore, the medical imageanalysis apparatus (200) can provide a consistent interpretation resultwithout relying on the subjective and variable DICOM header.

FIG. 10 is a flow diagram that illustrates process of detecting lesionaccording to an embodiment of this disclosure.

The medical image analysis apparatus (200) can receive the medical imageof the patient. As explained in FIG. 8, the medical image analysisapparatus (200) can predict metadata of the patient's medical imageusing the prediction model. Since metadata is predicted based on thesame prediction model, the medical image analysis apparatus (200) canacquire metadata based on the same criteria regardless of the medicalimage received. Therefore, the medical image analysis apparatus (200)can increase the success rate of the target task by applying machinelearning to the medical image and metadata. The target task can bedetection of lesion.

The medical image analysis apparatus (200) can execute a step (1010) inwhich the patient's medical image is adjusted based on predictedmetadata to detect anomaly in the input medical image. In addition, themedical image analysis apparatus (200) can execute a step in which atleast one of window center of the input medical image, window width,color, and output direction is adjusted.

The medical image analysis apparatus (200) can execute a step (1020) inwhich body parts included in predicted metadata are checked. The medicalimage analysis apparatus (200) can verify whether body part informationof predicted metadata agrees with the body part for which abnormality(anomaly) is to be detected.

For instance, the user may need a medical image for a specific body partto diagnose abnormality using the medical image of the patient. The usercan enter information about the specific body part into the medicalimage analysis apparatus (200). Otherwise, the medical image analysisapparatus (200) can automatically acquire information about the specificbody part that corresponds to lesion that the user is looking for. Themedical image analysis apparatus (200) can verify whether the patient'smedical image corresponds to the specific body part by comparinginformation about the specific body part with body part informationincluded in metadata. If information about the specific body part doesnot agree with body part information included in metadata, the medicalimage analysis apparatus (200) can acquire a new medical image of thepatient or execute operation to acquire a new medical image of thepatient.

The medical image analysis apparatus (200) can executed a step (1030) inwhich modality included in predicted metadata is checked. The medicalimage analysis apparatus (200) can verify whether modality informationof predicted metadata is appropriate for detecting anomaly.

For instance, the user may need a medical image of a specific modalityto diagnose abnormality using the medical image of the patient. The usercan enter information about the specific modality into the medical imageanalysis apparatus (200). Otherwise, the medical image analysisapparatus (200) can automatically acquire information about the specificmodality to detect lesion that the user is looking for. The medicalimage analysis apparatus (200) can verify whether the patient's medicalimage is based on the specific modality by comparing information aboutthe specific modality with modality information included in metadata. Ifinformation about the specific modality does not agree with modalityinformation included in metadata, the medical image analysis apparatus(200) can acquire a new medical image of the patient or executeoperation to acquire a new medical image of the patient.

The medical image analysis apparatus (200) can executed a step (1440) inwhich patient information included in predicted metadata is checked. Themedical image analysis apparatus (200) can verify whether patientinformation of predicted metadata is appropriate for detectingabnormality.

For instance, the user may need information about a specific patient todiagnose abnormality using the medical image of the patient. The medicalimage analysis apparatus (200) can decide whether the target ofdiagnosis is the same person as the patient of the medical image. Inaddition, the medical image analysis apparatus (200) can diagnose thepatient of specific age range to diagnose abnormality. For instance, theuser can enter patient information into the medical image analysisapparatus (200). Otherwise, the medical image analysis apparatus (200)can automatically acquire patient information to detect lesion that theuser is looking for. The medical image analysis apparatus (200) cancompare input patient information with patient information included inmetadata. If input patient information does not agree with patientinformation included in metadata, the medical image analysis apparatus(200) can display a warning message.

The medical image analysis apparatus (200) can execute a step (1050) inwhich lesion is detected from the medical image of the patient. Themedical image analysis apparatus (200) can use a machine learning modelspecialized in lesion detection to detect lesion from the medical image.

Various embodiments were examined so far. A person with common knowledgein the technical field of this invention would understand that thisinvention can be embodied into various other forms without deviatingfrom essential characteristics of this invention. Therefore, theembodiments disclosed must be considered from an explanatory perspectiveinstead of a limited perspective. Scope of this invention is shown inclaims instead of earlier explanation, and all differences within thisscope should be interpreted as to be included in this invention.

On the one hand, certain embodiments of this invention described abovecan be written as programs that can be executed on a PC, and they can beembodied on a general-purpose digital PC that operates the programsabove using computer readable recording media. Computer readablerecording media above include storage media such as magnetic storagemedia (for instance, ROM, floppy disk, hard disk, etc.) and opticalreading media (for instance, CD-ROM, DVD, etc.).

What is claimed is:
 1. A computerized medical image analysis method,using a hardware processor and a hardware memory, comprising:machine-training a prediction model for predicting metadata of an inputmedical image based on a plurality of image-metadata sets; processingthe input medical image to obtain predicted metadata of the inputmedical image, wherein the predicted metadata comprises at least one ofinformation related to one or more objects included in the input medicalimage, information about a shooting environment of the input medicalimage, information about patient corresponding to the input medicalimage, and information related to a display method of the input medicalimage; and processing the input medical image to detect at least one ofabnormality in the input medical image based on the predicted metadata,wherein the plurality of image-metadata sets comprise a plurality ofdigital imaging and communications in medicine (DICOM) files eachcomprising a DICOM image file and a DICOM header.
 2. The method of claim1, further comprising adjusting the input image using predictedmetadata, wherein the display method information comprises at least oneof window center information, window width information, color inversioninformation, image rotation information, and image flip information ofthe input medical image.
 3. The method of claim 2, wherein adjusting theinput medical image comprises adjusting at least one of a window center,a window width, a color, and an output direction of the input medicalimage based on the predicted metadata.
 4. The method of claim 1, whereinthe plurality of image-metadata sets comprise a training medical imageand metadata of the training medical data, and the input medical imageis obtained from an input DICOM tile comprising an input DICOM header ofthe input medical image.
 5. The method of claim 1, wherein the shootingenvironment of the input medical image includes modality information ofthe input medical image, and wherein the processing the input medicalimage to detect at least one abnormality comprises detecting at leastone of abnormality in the input medical image based on a predictionmodel corresponding to the modality information included in thepredicted metadata.
 6. The method of claim 1, further comprisingmatching and saving the predicted metadata of the input medical image inassociation with the input medical image.
 7. The method of claim 1,further comprising verifying whether body part information of thepredicted metadata matches a target body part for which abnormality isto be detected.
 8. The method of claim 1, further comprising verifyingwhether modality information of the predicted metadata is appropriatefor detecting anomaly.
 9. The method of claim 1, further comprisingverifying whether patient information of predicted metadata isappropriate for detecting abnormality.
 10. A medical image analysisapparatus comprising a memory storing computer-executable instructionsand a processor configured to execute the computer-executableinstructions, wherein the processor is configured, b executing thecomputer-executable instructions, to perform: machine-training aprediction model for predicting metadata of an input medical image basedon a plurality of image-metadata sets; processing the input medicalimage to obtain predicted metadata of the input medical image, whereinthe predicted metadata comprises at least one of information related toone or more objects included in the input medical image, informationabout a shooting environment of the input medical image, informationabout patient corresponding to the input medical image, and informationrelated to a display method of the input medical image; and processingthe input medical image to detect at least one of abnormality in theinput medical image based on the predicted metadata, wherein theplurality of image-metadata sets comprise a plurality of digital imagingand communications in medicine (DICOM) files each comprising a DICOMimage file and a DICOM header.
 11. The apparatus of claim 10, whereinthe processor is further configured, by executing thecomputer-executable instructions, to perform adjusting the input imageusing predicted metadata, and wherein the display method informationcomprises at least one of window center information, window widthinformation, color inversion information, image rotation information,and image flip information of the input medical image.
 12. The apparatusof claim 11, wherein the processor is further configured to performadjusting the input medical image comprises adjusting at least one of awindow center, a window width, a color, and an output direction of theinput medical image based on the predicted metadata.
 13. The apparatusof claim 10, wherein the plurality of image-metadata sets comprise atraining medical image and metadata of the training medical data, andthe input medical image is obtained from an input DICOM file comprisingan input DICOM header of the input medical image.
 14. The apparatus ofclaim 10, wherein the shooting environment of the input medical imageincludes modality information of the input medical image, and whereinthe processing the input medical image to detect at least oneabnormality comprises detecting at least one of abnormality in the inputmedical image based on a prediction model corresponding to the modalityinformation included in the predicted metadata.
 15. The apparatus ofclaim 10, wherein the processor is further configured, by executing thecomputer-executable instructions, to perform matching and saving thepredicted metadata of the input medical image in association with theinput medical image.
 16. The apparatus of claim 10, wherein theprocessor is further configured, by executing the computer-executableinstructions, to perform verifying whether body part information of thepredicted metadata matches a target body part for which abnormality isto be detected.
 17. The apparatus of claim 10, wherein the processor isfurther configured, by executing the computer-executable instructions,to perform verifying whether modality information of the predictedmetadata is appropriate for detecting anomaly.
 18. The apparatus ofclaim 10, wherein the processor is further configured, by executing thecomputer-executable instructions, to perform verifying whether patientinformation of predicted metadata is appropriate for detectingabnormality.